all keys are still present on versioned remote after import of a tree

When importing from versioned remotes, fix tracking of the content of
deleted files.

Only S3 supports versioning so far, so only it was affected.

But, the draft import/export interface for external remotes also seemed to
need a change, so that versionedExport could be set.

diff --git a/Annex/Import.hs b/Annex/Import.hs
index 1ab5c01abab4904fd59f39a22dfcf91a00a7ab36..c16eb1821382ebd2a6582b30278be445600a12a3 100644 (file)
--- a/Annex/Import.hs
+++ b/Annex/Import.hs
@@ -184,10 +184,13 @@ recordImportTree remote importtreeconfig importable = do
                                unlessM (stillpresent db oldkey) $
                                        logChange oldkey (Remote.uuid remote) InfoMissing
                        _ -> noop
-               db <- Export.openDb (Remote.uuid remote)
-               forM_ (exportedTreeishes oldexport) $ \oldtree ->
-                       Export.runExportDiffUpdater updater db oldtree finaltree
-               Export.closeDb db
+               -- When the remote is versioned, it still contains keys
+               -- that are not present in the new tree.
+               unless (Remote.versionedExport (Remote.exportActions remote)) $ do
+                       db <- Export.openDb (Remote.uuid remote)
+                       forM_ (exportedTreeishes oldexport) $ \oldtree ->
+                               Export.runExportDiffUpdater updater db oldtree finaltree
+                       Export.closeDb db
 
 buildImportCommit' :: Remote -> ImportCommitConfig -> Maybe Sha -> History Sha -> Annex (Maybe Sha)
 buildImportCommit' remote importcommitconfig mtrackingcommit imported@(History ti _) =

diff --git a/CHANGELOG b/CHANGELOG
index c393dad10169246e3d8fe444836ceed2aec9ba71..56f2c781fe83f3730a91162f881451172fac83a8 100644 (file)
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -6,8 +6,10 @@ git-annex (10.20221004) UNRELEASED; urgency=medium
     do not operate on a repository that has an empty name.
   * move: Fix openFile crash with -J
     (Fixes a reversion in 8.20201103)
-  * S3: Speed up importing from a large bucket when fileprefix= is set
+  * S3: Speed up importing from a large bucket when fileprefix= is set,
     by only asking for files under the prefix.
+  * When importing from versioned remotes, fix tracking of the content
+    of deleted files.
 
  -- Joey Hess <id@joeyh.name>  Mon, 03 Oct 2022 13:36:42 -0400
 

diff --git a/Command/Export.hs b/Command/Export.hs
index 3c5b3ad21409f8ff3403a987109c16c799617eff..d331bb3030cde168da6120e513951dd4d136ba27 100644 (file)
--- a/Command/Export.hs
+++ b/Command/Export.hs
@@ -378,7 +378,7 @@ cleanupUnexport r db eks loc = do
                        removeExportedLocation db ek loc
                flushDbQueue db
 
-       -- An versionedExport remote supports removeExportLocation to remove
+       -- A versionedExport remote supports removeExportLocation to remove
        -- the file from the exported tree, but still retains the content
        -- and allows retrieving it.
        unless (versionedExport (exportActions r)) $ do

diff --git a/Types/Remote.hs b/Types/Remote.hs
index e9e8a8c815f7dda44acc424d947df396f551010f..121d4fd5cdbdcef1c9740794bcf463b8c7678b07 100644 (file)
--- a/Types/Remote.hs
+++ b/Types/Remote.hs
@@ -270,9 +270,10 @@ data ExportActions a = ExportActions
        -- Can throw exception if unable to access remote, or if remote
        -- refuses to remove the content.
        , removeExport :: Key -> ExportLocation -> a ()
-       -- Set when the content of a Key stored in the remote to an
-       -- ExportLocation and then removed with removeExport remains
-       -- accessible to retrieveKeyFile and checkPresent.
+       -- Set when the remote is versioned, so once a Key is stored
+       -- to an ExportLocation, a subsequent deletion of that
+       -- ExportLocation leaves the key still accessible to retrieveKeyFile
+       -- and checkPresent.
        , versionedExport :: Bool
        -- Removes an exported directory. Typically the directory will be
        -- empty, but it could possibly contain files or other directories,

diff --git a/doc/bugs/annex_import_does_not_account_for_versioning_on_S3.mdwn b/doc/bugs/annex_import_does_not_account_for_versioning_on_S3.mdwn
index d631fac1770ff7080e72cdbebf0e2edef2f24f66..4f6229e34cb0824b4657848e061f19a4d2e1ed25 100644 (file)
--- a/doc/bugs/annex_import_does_not_account_for_versioning_on_S3.mdwn
+++ b/doc/bugs/annex_import_does_not_account_for_versioning_on_S3.mdwn
@@ -13,3 +13,4 @@ Observed with several versions from 8-10 on linux. As laid out above, I strongly
 
 Lots. I love git-annex.
 
+> [[fixed|done]] --[[Joey]]

diff --git a/doc/bugs/annex_import_does_not_account_for_versioning_on_S3/comment_1_a04fbb936785456ed99512ea4c29fd53._comment b/doc/bugs/annex_import_does_not_account_for_versioning_on_S3/comment_1_a04fbb936785456ed99512ea4c29fd53._comment
new file mode 100644 (file)
index 0000000..c3d8aa0
--- /dev/null
+++ b/doc/bugs/annex_import_does_not_account_for_versioning_on_S3/comment_1_a04fbb936785456ed99512ea4c29fd53._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2022-10-11T16:28:39Z"
+ content="""
+This looks like a simple fix. After importing from a versioned remote,
+it can just skip updating the location logs to remove the keys that are not
+present in the current tree. The same as is already done when exporting
+to a versioned remote. I've made that change.
+"""]]

diff --git a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
index 24004402fe0eb93c00f50985ac5c60a7a8efa866..b2f07bad0b5876a3a0dbceb5c4367c594290a141 100644 (file)
--- a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
+++ b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
@@ -150,6 +150,13 @@ support a request, it can reply with `UNSUPPORTED-REQUEST`.
     Indicates that `IMPORTKEY` can be used.
   * `IMPORTKEYSUPPORTED-FAILURE`  
     Indicates that `IMPORTKEY` cannot be used.
+ * `VERSIONED`
+  Used to check if the special remote is versioned.
+  Note that this request may be made before or after `PREPARE`.
+  * `ISVERSIONED`  
+    Indicates that the remote is versioned.
+  * `NOTVERSIONED`  
+    Indicates that the remote is not versioned.
 * `LISTIMPORTABLECONTENTS`  
   Used to get a list of all the files that are stored in the special 
   remote. A block of responses
@@ -170,6 +177,8 @@ support a request, it can reply with `UNSUPPORTED-REQUEST`.
     be nested multiple levels deep.  
     This should only be used when the remote supports using
     "TRANSFER RECEIVE Key" to retrieve historical versions of files.
+    And, it should only be used when the remote replies `ISVERSIONED`
+    to the `VERSIONED` message.
   * `END`  
     Indicates the end of a block of responses.
 * `LOCATION Name`